{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FEATURE EXTRACTION WITH DATA AUGMENTATION and Fine-tuning\n",
    "1 Add your custom network on top of an already-trained base network.\n",
    "\n",
    "2 Freeze the base network.\n",
    "\n",
    "3 Train the part you added.\n",
    "\n",
    "4 Unfreeze some layers in the base network.\n",
    "\n",
    "5 Jointly train both these layers and the part you added.\n",
    "\n",
    "You already completed the first three steps when doing feature extraction. Let’s proceed with step 4: you’ll unfreeze your conv_base and then freeze individual layers\n",
    "inside it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Description of VGG16 model\n",
    "VGG16 (also called OxfordNet) is a convolutional neural network architecture named after the Visual Geometry Group from Oxford, who developed it.\n",
    "It was used to win the ILSVR (ImageNet) competition in 2014.\n",
    "To this day is it still considered to be an excellent vision model, although it has been somewhat outperformed by more revent advances such as Inception and ResNet.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.applications import VGG16\n",
    "conv_base = VGG16(weights='imagenet',\n",
    "                  include_top=False,\n",
    "                  input_shape=(150, 150, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "vgg16 (Model)                (None, 4, 4, 512)         14714688  \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 8192)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 256)               2097408   \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 16,812,353\n",
      "Trainable params: 16,812,353\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 1 Add your custom network on top of an already-trained base network.\n",
    "\n",
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(conv_base)\n",
    "model.add(layers.Flatten())\n",
    "model.add(layers.Dense(256, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cats_and_dogs_small\\train\n",
      "cats_and_dogs_small\\validation\n",
      "cats_and_dogs_small\\test\n"
     ]
    }
   ],
   "source": [
    "original_dataset_dir = \"train\"\n",
    "base_dir = 'cats_and_dogs_small'\n",
    "train_dir = os.path.join(base_dir, 'train')\n",
    "validation_dir = os.path.join(base_dir, 'validation')\n",
    "test_dir = os.path.join(base_dir, 'test')\n",
    "print(train_dir)\n",
    "print(validation_dir)\n",
    "print(test_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.freezing the convolutional base\n",
    "In Keras, you freeze a network by setting its trainable attribute to False:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the number of trainable weights before freezing the conv base: 30\n"
     ]
    }
   ],
   "source": [
    "print('This is the number of trainable weights ''before freezing the conv base:', len(model.trainable_weights))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the number of trainable weights after freezing the conv base: 4\n"
     ]
    }
   ],
   "source": [
    "conv_base.trainable = False\n",
    "print('This is the number of trainable weights ''after freezing the conv base:', len(model.trainable_weights))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training the model end to end with a frozen convolutional base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "train_datagen = ImageDataGenerator(\n",
    "                rescale=1./255,\n",
    "                rotation_range=40,\n",
    "                width_shift_range=0.2,\n",
    "                height_shift_range=0.2,\n",
    "                shear_range=0.2,\n",
    "                zoom_range=0.2,\n",
    "                horizontal_flip=True,\n",
    "                fill_mode='nearest')\n",
    "\n",
    "test_datagen = ImageDataGenerator(rescale=1./255)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2000 images belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "train_generator = train_datagen.flow_from_directory(\n",
    "                    train_dir,\n",
    "                    target_size=(150, 150),\n",
    "                    batch_size=20,\n",
    "                    class_mode='binary')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1000 images belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "validation_generator = test_datagen.flow_from_directory(\n",
    "                        validation_dir,\n",
    "                        target_size=(150, 150),\n",
    "                        batch_size=20,\n",
    "                        class_mode='binary')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import optimizers\n",
    "model.compile(loss='binary_crossentropy',\n",
    "                optimizer=optimizers.RMSprop(lr=2e-5),\n",
    "                metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.Train the part you added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "2/2 [==============================] - 23s 11s/step - loss: 0.7561 - acc: 0.4250 - val_loss: 0.7179 - val_acc: 0.4670\n",
      "Epoch 2/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6865 - acc: 0.5250 - val_loss: 0.6997 - val_acc: 0.5100\n",
      "Epoch 3/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6965 - acc: 0.5250 - val_loss: 0.6853 - val_acc: 0.5480\n",
      "Epoch 4/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6359 - acc: 0.6250 - val_loss: 0.6820 - val_acc: 0.5480\n",
      "Epoch 5/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6571 - acc: 0.6000 - val_loss: 0.6677 - val_acc: 0.5780\n",
      "Epoch 6/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6724 - acc: 0.5500 - val_loss: 0.6799 - val_acc: 0.5370\n",
      "Epoch 7/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6608 - acc: 0.6250 - val_loss: 0.6687 - val_acc: 0.5570\n",
      "Epoch 8/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6712 - acc: 0.5000 - val_loss: 0.6509 - val_acc: 0.6140\n",
      "Epoch 9/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6788 - acc: 0.5500 - val_loss: 0.6398 - val_acc: 0.6600\n",
      "Epoch 10/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.7103 - acc: 0.5250 - val_loss: 0.6301 - val_acc: 0.6810\n",
      "Epoch 11/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6300 - acc: 0.6750 - val_loss: 0.6274 - val_acc: 0.6720\n",
      "Epoch 12/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.7069 - acc: 0.4500 - val_loss: 0.6115 - val_acc: 0.7060\n",
      "Epoch 13/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6601 - acc: 0.6000 - val_loss: 0.6054 - val_acc: 0.7010\n",
      "Epoch 14/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6364 - acc: 0.6750 - val_loss: 0.6144 - val_acc: 0.6460\n",
      "Epoch 15/30\n",
      "2/2 [==============================] - 15s 8s/step - loss: 0.6734 - acc: 0.6000 - val_loss: 0.5954 - val_acc: 0.7210\n",
      "Epoch 16/30\n",
      "2/2 [==============================] - 16s 8s/step - loss: 0.6112 - acc: 0.7250 - val_loss: 0.6113 - val_acc: 0.6370\n",
      "Epoch 17/30\n",
      "2/2 [==============================] - 17s 9s/step - loss: 0.6588 - acc: 0.6750 - val_loss: 0.5851 - val_acc: 0.7380\n",
      "Epoch 18/30\n",
      "2/2 [==============================] - 17s 9s/step - loss: 0.6677 - acc: 0.5500 - val_loss: 0.5760 - val_acc: 0.7520\n",
      "Epoch 19/30\n",
      "2/2 [==============================] - 17s 9s/step - loss: 0.6162 - acc: 0.7500 - val_loss: 0.5716 - val_acc: 0.7560\n",
      "Epoch 20/30\n",
      "2/2 [==============================] - 17s 9s/step - loss: 0.6299 - acc: 0.6250 - val_loss: 0.5642 - val_acc: 0.7770\n",
      "Epoch 21/30\n",
      "2/2 [==============================] - 18s 9s/step - loss: 0.5991 - acc: 0.7250 - val_loss: 0.5597 - val_acc: 0.7680\n",
      "Epoch 22/30\n",
      "2/2 [==============================] - 22s 11s/step - loss: 0.5795 - acc: 0.7250 - val_loss: 0.5587 - val_acc: 0.7630\n",
      "Epoch 23/30\n",
      "2/2 [==============================] - 23s 11s/step - loss: 0.6230 - acc: 0.6750 - val_loss: 0.5516 - val_acc: 0.7720\n",
      "Epoch 24/30\n",
      "2/2 [==============================] - 22s 11s/step - loss: 0.5885 - acc: 0.6750 - val_loss: 0.5489 - val_acc: 0.7750\n",
      "Epoch 25/30\n",
      "2/2 [==============================] - 22s 11s/step - loss: 0.6021 - acc: 0.6750 - val_loss: 0.5412 - val_acc: 0.7810\n",
      "Epoch 26/30\n",
      "2/2 [==============================] - 20s 10s/step - loss: 0.6027 - acc: 0.7250 - val_loss: 0.5372 - val_acc: 0.7900\n",
      "Epoch 27/30\n",
      "2/2 [==============================] - 18s 9s/step - loss: 0.5504 - acc: 0.8000 - val_loss: 0.5303 - val_acc: 0.7870\n",
      "Epoch 28/30\n",
      "2/2 [==============================] - 18s 9s/step - loss: 0.6142 - acc: 0.6750 - val_loss: 0.5331 - val_acc: 0.7660\n",
      "Epoch 29/30\n",
      "2/2 [==============================] - 18s 9s/step - loss: 0.5860 - acc: 0.7000 - val_loss: 0.5257 - val_acc: 0.7850\n",
      "Epoch 30/30\n",
      "2/2 [==============================] - 18s 9s/step - loss: 0.5047 - acc: 0.8500 - val_loss: 0.5178 - val_acc: 0.8040\n"
     ]
    }
   ],
   "source": [
    "history = model.fit_generator(\n",
    "                train_generator,\n",
    "                steps_per_epoch=2, #100\n",
    "                epochs=30,\n",
    "                validation_data=validation_generator,\n",
    "                validation_steps=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the number of trainable weights after freezing the conv base: 4\n"
     ]
    }
   ],
   "source": [
    "print('This is the number of trainable weights ''after freezing the conv base:', len(model.trainable_weights))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#conv_base.trainable = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print('This is the number of trainable weights ''after setting conv base to trainable:', len(model.trainable_weights))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.Unfreeze some layers in the base network.\n",
    "Freezing all layers up to a specific one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_trainable = False\n",
    "for layer in conv_base.layers:\n",
    "    if layer.name == 'block5_conv1':\n",
    "        set_trainable = True\n",
    "    if set_trainable:\n",
    "        layer.trainable = True\n",
    "    else:\n",
    "        layer.trainable = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the number of trainable weights after Un-freezing some layers of conv base: 4\n"
     ]
    }
   ],
   "source": [
    "print('This is the number of trainable weights ''after Un-freezing some layers of conv base:', len(model.trainable_weights))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "optimizer=optimizers.RMSprop(lr=1e-5),\n",
    "metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "100/100 [==============================] - 103s 1s/step - loss: 0.5358 - acc: 0.7505 - val_loss: 0.4353 - val_acc: 0.8340\n",
      "Epoch 2/2\n",
      "100/100 [==============================] - 102s 1s/step - loss: 0.4897 - acc: 0.7870 - val_loss: 0.3907 - val_acc: 0.8530\n"
     ]
    }
   ],
   "source": [
    "epochs = 2 # 100\n",
    "history = model.fit_generator(\n",
    "                train_generator,\n",
    "                steps_per_epoch=100,\n",
    "                epochs=epochs,\n",
    "                validation_data=validation_generator,\n",
    "                validation_steps=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_gpu",
   "language": "python",
   "name": "conda_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
